Introduction to Machine Learning

Chapter 02: CRISP-ML(Q), Preprocessing and Exploratory Data Analysis

1. Introduction

This unit covers the critical early stages of a machine learning project: the methodology framework, exploratory data analysis (EDA), data cleaning, categorical encoding, and feature scaling. These steps typically consume 60–80% of project time, and they set an upper limit on how well the later stages can perform. A model trained on poorly prepared data will not give reliable results, no matter which algorithm is used. By the end of this unit, you will be able to take raw, messy real-world data and prepare it correctly for downstream machine learning algorithms.

Learning Objectives
Name and explain the six phases of the CRISP-ML(Q) lifecycle
Distinguish nominal, binary, ordinal, and numerical feature types
Handle missing values via removal or imputation (mean, KNN)
Encode categorical variables using one-hot, label, and ordinal encoding
Apply Min-Max normalization and Z-score standardization and explain when to prefer each
Use automated EDA tools (ydata-profiling, AutoViz) alongside manual pandas/seaborn exploration

2. Theory

2.1 CRISP-ML(Q) — Six Lifecycle Phases

Real machine learning projects follow a structured methodology rather than applying algorithms directly to whatever data happens to be available. CRISP-ML(Q) is a widely used framework of this kind. It divides a project into six phases:

1. Business & Data

2. Data Engineering

3. Modeling

4. QA / Evaluation

5. Deployment

6. Monitoring

Business and Data Understanding

This phase ensures project feasibility before significant resources are committed. The main tasks include identifying the ML application scope and business success criteria, defining measurable KPIs, assessing the availability of time, technology, and human resources, and verifying that sufficient high-quality data exists. If the data is inadequate, the team may need to redesign the data collection approach. For example, consider a spam detection project. The business success criterion might be "build an effective spam filter." The KPIs could be specified as "achieve 95% accuracy with less than 1% false positive rate on held-out validation data." These measurable targets allow the team to determine whether the model meets the business requirements.

Data Engineering (Data Preparation)

This is the phase where the current unit primarily lives. Tasks include data selection and discarding low-quality samples, cleaning missing values and outliers, feature engineering (creating derived features), encoding categorical variables, and applying standardization or normalization. This phase often takes 60–80% of the total project time. Skipping it properly will ruin every downstream model, regardless of which algorithm you choose.
A critical rule applies to several of these tasks: any operation that learns parameters from the data (such as the mean for imputation, category mappings for encoding, or min/max values for scaling) must be performed after the train/test split. These parameters must be learned exclusively from the training set and then applied to the test set. Performing these operations on the entire dataset before splitting causes data leakage, which will make your evaluation results misleadingly optimistic. We will highlight this requirement in the specific sections below.

ML Model Engineering

In this phase, you apply the algorithms you will learn throughout this course. The work involves translating the business problem into a specific ML task such as classification, regression, or clustering. You then perform model selection, specialization, and training. It is also essential to collect metadata about the experiment, including the algorithm used, train/validation/test splits, hyperparameters, and the runtime environment. This phase often requires stepping back to the data engineering phase for additional feature work.

Quality Assurance

This phase involves offline testing on a held-out test set. You validate model performance against the KPIs defined in Phase 1 and analyze whether the business objectives will actually be achieved. Every evaluation outcome must be carefully documented to support decision-making about whether the model is ready for production.

Deployment

Deployment exposes the model to real users. This can take many forms, such as interactive dashboards, pre-computed predictions, plug-in components, or web service API endpoints. You must also define the update and retraining process. Examples of deployment include a Flask/FastAPI REST endpoint, mobile-app integration, or an automated weekly report pipeline.

Monitoring and Maintenance

Model performance tends to decay over time due to "data drift" or "concept drift" — changes in the data distribution or the underlying relationships in the problem. Therefore, you must track live prediction quality and distribution shifts. The model should be retrained on a schedule or whenever KPI thresholds are breached. Additionally, every prediction and its corresponding ground truth should be logged to support post-hoc audits and future improvements.

2.2 Feature / Variable Types

Before preprocessing data, it is important to understand the type of each feature. Different feature types require different preprocessing and encoding methods.

Type	Description	Examples
Nominal (Categorical)	Unordered categories	Hair color, marital status, customer_id
Binary	Two nominal categories	is_smoker, medical_test_result (+ve / −ve)
Ordinal	Ordered categories	Shirt size (S/M/L/XL), grades, customer_satisfaction
Numerical	Continuous or discrete numbers	Age, temperature, salary, number_of_dependents

2.3 Handling Missing Values

Missing values are common in real-world datasets. There are two general approaches: removing the affected rows or columns, or imputing a replacement value. The choice depends on how much data is missing and whether the missing values are informative. The two tabs below show how each approach is applied in pandas:

Removal

Imputation

Removal — The simplest approach is to remove the offending rows or columns. Pandas provides several operations for this purpose:

df.dropna(axis=0)       # remove rows with any missing 

df.dropna(axis=1)       # remove cols with any missing 

df.dropna(how='all')    # remove rows where ALL columns are NaN 

df.dropna(thresh=4)     # drop rows with fewer than 4 real values 

df.dropna(subset=['C']) # only drop rows where column C is NaN

However, overusing removal can result in the loss of valuable data. Removing rows is generally reasonable only when a very small percentage of rows contain missing values. Columns should usually be removed only when a large proportion of their values are missing or the feature is not useful.

Imputation replaces missing values with estimated values. Common imputation methods include replacing missing numerical values with the mean, median, or mode of the column, or using more sophisticated techniques like KNN imputation. We will explore these methods in more detail in later units.

When using imputation, the imputation values — for example, the mean or median of a column — must be computed using the training set only. These computed values are then used to fill missing values in both the training set and the test set. This ensures that information from the test set does not influence the training process. If you calculate the mean on the full dataset before splitting, you have introduced data leakage.

2.4 Encoding Categorical Variables

Most ML algorithms work with numerical input rather than categorical values represented as text. Therefore, categorical variables need to be converted into numerical representations. Three commonly used techniques are one-hot encoding, ordinal encoding, and label encoding.

One-Hot Encoding

Create a new binary dummy feature for each unique value in the original categorical feature. For a color feature with values {blue, green, red}:

Original color	blue	green	red
blue	1	0	0
green	0	1	0
red	0	0	1

pd.get_dummies(df, columns=['color'])

One-hot encoding is suitable for nominal features where there is no natural ordering. Be aware that this approach increases the number of columns in your dataset by the number of unique categories minus one (to avoid multicollinearity).

Ordinal Encoding

Ordinal encoding is used when the categories have a meaningful order. The desired ordering can be specified explicitly.

desired_order = ["S", "M", "L", "XL"] 

		encoder = OrdinalEncoder(categories=[desired_order]) 

		df["shirt_size"] = encoder.fit_transform(df["shirt_size"].values.reshape(-1, 1))

Label Encoding

Label encoding is used for the target class labels (y), not for input features. It maps class names to integer values such as 0, 1, 2, and so on.

class_le = LabelEncoder() 

df['classlabel'] = class_le.fit_transform(df['classlabel'].values)

Label encoding should not be used for input features unless the feature is ordinal, because the arbitrary integer assignment might imply an ordering that does not actually exist.

2.5 Feature Scaling: Normalization vs. Standardization

Feature scaling is important when features have very different numerical ranges. For example, if Age ranges from 18–65 while Salary ranges from $20k–$300k, distance-based algorithms such as kNN, SVM, and clustering can allow Salary to dominate the distance calculation. Scaling puts features on comparable scales so that differences in their original numerical ranges do not have an undue effect on the model.

Min-Max Normalization (bounded interval)

Scales every feature to [0, 1] using the per-feature min and max:

$ x'_{\text{norm}} = \frac{x - x_{\min}}{x_{\max} - x_{\min}} $

Z-Score Standardization (zero mean, unit variance)

Centers each feature column at mean 0 with standard deviation 1 (parameters of the standard normal distribution):

$ x'_{\text{std}} = \frac{x - \mu_x}{\sigma_x} $

Standardization does not change the shape of a distribution, nor convert a non-normal distribution to normal.

Situation	Prefer Normalization	Prefer Standardization
Distance-based algorithms (kNN, clustering)	✅ Usually	Also valid
Bounded output range needed (e.g., image pixels)	✅ Always	—
Outliers present in the data	—	✅ Less sensitive
Neural Networks or PCA	—	✅ Required / Preferred
Gradient-descent optimization (e.g., LogReg)	—	✅ Usually

2.6 Exploratory Data Analysis (EDA) Tools

Before preparing features for downstream modeling, we first need to understand what is present in the raw data. Exploratory Data Analysis (EDA) helps identify issues such as missing values, outliers, imbalance, and correlations.

EDA can be performed through two complementary tracks: manual EDA and automated EDA.

Manual EDA

Manual exploration can be performed using:

pandas: .describe(), .info(), .isna()
matplotlib: bar plots, histograms, scatter plots, and box plots
seaborn: heatmaps

Automated EDA

Automated tools can provide a broader initial overview of the dataset. Examples include:

ydata-profiling
AutoViz

Whichever track is used, the findings from EDA feed directly into the preprocessing decisions covered in the previous sections. The overall flow is:

            The techniques discussed so far including, handling missing values, encoding 
			categorical variables, and scaling features, are different parts of data 
			preparation within the Data Engineering phase of CRISP-ML(Q). In practice, 
			the appropriate steps depend on the characteristics of the dataset and the 
			ML algorithm being used. An important principle is that preprocessing parameters 
			should be learned from the training data and then applied consistently to 
			other data, such as the test set, to avoid data leakage.
			
        

3. Interactive Examples

Example 1: Identify the Feature Type

For each real-world feature, classify it as Nominal / Binary / Ordinal / Numerical. Click the reveal button below each scenario.

Scenario A: A "Level_of_Education" column with values {High School, Bachelor, Masters, PhD}.

Reveal classification & reasoning

Ordinal. The four values have a clear ordering from less to more education, so we should preserve that order with OrdinalEncoder rather than lose it to one-hot dummies.

Scenario B: A "State_of_Residence" column with values {CA, TX, NY, FL, …}.

Reveal classification & reasoning

Nominal (categorical). No ordering between states — one-hot encoding is appropriate.

Scenario C: An "Annual_Income_USD" column storing exact salaries.

Reveal classification & reasoning

Numerical (continuous). Must be scaled (standardization or normalization) before any distance-based model.

Example 2: To Scale or Not? And Which Scaler?

Decision Challenge

For each case below, decide whether scaling is required, and between Min-Max vs. Z-score standardization. Reveal each answer separately.

Case 1: Training a kNN classifier on features Age (years), Income (USD), and Height (cm).

Required: YES. kNN is distance-based and Income would dominate. Prefer Z-score standardization if outliers are likely (outliers distort min/max). Prefer Min-Max if the dataset is known to be clean with natural bounded ranges.

Case 2: Training a Random Forest classifier on the same three features.

Required: NO. Tree-based models (Decision Tree, Random Forest, Gradient Boosting) are scale-invariant because they split on threshold-per-feature rather than compare across features via distances. Scaling won't hurt but wastes CPU cycles.

Case 3: Feeding features into a PCA dimensionality-reduction step before classification.

Required: YES, always standardize before PCA. PCA looks for directions of maximum variance; unscaled features make the high-variance-but-arbitrary-scale columns (e.g., income in cents vs. dollars) dominate all principal components.

Example 3: CRISP-ML(Q) Phase Matching

Click to show 3 activities. Match each activity to the correct CRISP-ML(Q) phase.

"Team defines KPIs and checks whether 3 months of labeled data exists."
"DevOps deploys the final model as a FastAPI endpoint behind a load balancer."
"Data scientist drops duplicate rows, replaces NaN incomes with medians, and one-hot encodes 'job_type'."

→ Phase 1 — Business and Data Understanding (feasibility + KPI definition)
→ Phase 5 — Deployment (exposing predictions to production users)
→ Phase 2 — Data Engineering (cleaning + encoding)

4. Numerical Solutions

Problem 1: Min-Max Normalization

Given the ages {26, 28, 34, 38}: normalize each value using Min-Max scaling so the outputs lie in [0, 1].

📘 Step-by-step solution

Step 1: Identify the range.

$ x_{\min} = 26,\quad x_{\max} = 38 $

Step 2: Apply the formula to each point.

$ x' = \dfrac{x - 26}{38 - 26} = \dfrac{x - 26}{12} $

Raw age	Calculation	Normalized x'
26	(26−26)/12	0.000
28	(28−26)/12	0.167
34	(34−26)/12	0.667
38	(38−26)/12	1.000

Check: min maps to 0, max maps to 1 ✓.

Problem 2: Z-Score Standardization

Given salaries {$100,000; $140,000; $150,000; $300,000}. Compute the sample mean, sample standard deviation, and then the Z-score for each value.

📘 Step-by-step solution

Step 1: Compute sample mean.

$ \mu = \dfrac{100 + 140 + 150 + 300}{4} = \dfrac{690}{4} = \mathbf{172.5} $ thousand dollars.

Step 2: Compute sample standard deviation (divide by n−1).

$ \sigma = \sqrt{\dfrac{(100{-}172.5)^2 + (140{-}172.5)^2 + (150{-}172.5)^2 + (300{-}172.5)^2}{3}} $

$ = \sqrt{\dfrac{5256.25 + 1056.25 + 506.25 + 16256.25}{3}} = \sqrt{7691.\overline{6}} \approx \mathbf{87.70} $ thousand.

Step 3: Apply Z = (x − μ) / σ per salary.

Salary ($K)	Z
100	−0.827
140	−0.370
150	−0.257
300	+1.454

Check: Mean of Z-scores is 0; sample SD ≈ 1 ✓. Notice the $300K salary pulls the mean up and the Z of +1.45 indicates it is not a massive outlier despite looking like one — a strength of standardization.

5. Try It Yourself

Problem 1 — Scaling Drill

You are given 5 test scores: {55, 62, 70, 78, 95}.

Compute their Min-Max normalized values.
Compute their Z-scores using sample mean and sample SD.
For each value, compare the two scaled values. Why is the Min-Max of 70 larger than the Z-score of 70?

(a) Min=55, Max=95, range=40.

x	Min-Max
55	0.000
62	0.175
70	0.375
78	0.575
95	1.000

(b) μ = 72, σ ≈ 15.427.

x	Z
55	−1.10
62	−0.65
70	−0.13
78	+0.39
95	+1.49

(c) Min-Max of 70 is 0.375 because 70 sits 37.5% of the way from 55 to 95. The Z of 70 is negative because 70 is slightly below the mean of 72. The two scalers answer different questions: "position inside observed range" vs. "deviation from mean in SD units."

Problem 2 — Encoding Decisions

A dataset has the following features. For each, name the correct encoding strategy and justify in one sentence.

country_of_birth — 42 unique country names.
satisfaction_rating — "Very Unsatisfied" / "Unsatisfied" / "Neutral" / "Satisfied" / "Very Satisfied".
customer_churn — target variable "Stayed" / "Churned".
monthly_charges_usd — continuous dollar amounts.

One-Hot encoding (pd.get_dummies or sklearn OneHotEncoder). Nominal; no ordering between 42 countries.
OrdinalEncoder with the explicit 5-level order. Must preserve the scale from Very Unsatisfied → Very Satisfied.
LabelEncoder (only the y target, never an X feature). Maps {Stayed, Churned} → {0, 1}.
No encoding needed. Numerical; do scale (Z-score or Min-Max) depending on the downstream model.

Problem 3 — Missing-Value Strategy

A dataset of 5,000 patients has the following missingness patterns. Recommend a concrete handling strategy for each column:

patient_zip_code: 47% missing.
resting_blood_pressure: 1.2% missing; no obvious pattern of missingness.
has_diabetes: 3.1% missing. You suspect diabetic patients forgot to tick the "Yes" box more often than non-diabetics did.

Drop the column entirely (axis=1). Almost half the data is missing — imputation would invent nearly half the column, and row-wise removal would lose half the dataset.
SimpleImputer with median (mean is okay too; median is more outlier-resistant). 1.2% is small, no pattern → impute with the marginal distribution center.
(i) Do a separate Missing Indicator column "is_diabetes_missing" + (ii) impute the missing entries. Missingness is MNAR (Missing Not At Random) related to the true value, so we must keep missingness itself as a feature. A KNN or MICE imputer would be smarter than the mean here.

6. Interactive Quiz

Answer all 5 questions. Click an option for instant feedback.

Your score: 0 / 5

7. Key Takeaways

CRISP-ML(Q) = 6 phases: Business & Data Understanding → Data Engineering → Model Engineering → QA/Evaluation → Deployment → Monitoring. Most of this course focuses on phases 2–4.
60–80% of ML work = Data Engineering. Every hour spent on cleaning, encoding, and scaling prevents days of debugging model pathology later.
Know your feature types. The four types (nominal / binary / ordinal / numerical) dictate every preprocessing decision — including whether to encode at all, which encoder to pick, and whether to scale.
Removal vs. Imputation. Drop rows or columns only when missingness is tiny; otherwise impute. Mean/median imputation is a fast baseline; KNN and MICE imputers preserve multivariate structure.
Min-Max (normalization) vs. Z-score (standardization). Use normalization when you need bounded outputs (images, neural nets). Use standardization when data has outliers or when feeding PCA, gradient-descent, or SVMs. Always scale before distance-based algorithms.
EDA is a prerequisite for cleaning. Automated tools (ydata-profiling, AutoViz) give the quick overview; manual pandas/seaborn work answers the targeted questions that automated reports miss.
Data leakage reminder. Every imputer, encoder, and scaler must be fit on the training set only and then the learned parameters applied to the test set.

8. Common Pitfalls

Encoding ordinal features with one-hot. Shirt sizes S/M/L/XL are not four independent categories. Dumping the order throws away information and hurts tree-less models.
LabelEncoding the X features (instead of only y). Assigning 0/1/2/3/4 to five unordered countries injects a fake "Canada < Germany" ordering. Use OneHotEncoder for nominal X features.
Dropping rows first and asking questions later. Look at why values are missing. Random 0.5% missingness is fine to drop; 20% missing that correlates with the target variable needs careful imputation + indicator features.
Fitting StandardScaler on the entire dataset before train/test split. Classic data leakage. The test set mean/std leaks into training. Fix: fit(train) then transform(train) and transform(test) separately.
Assuming trees need scaling. Scaling a Random Forest's inputs wastes compute and readability. Save it for distance-based, gradient-based, and PCA pipelines.
One-hot when you meant target-encoding a high-cardinality column. 42 countries → 41 extra dummy columns may blow up dimensionality. Consider dimensionality reduction (Chapters 5–6) after one-hot, or target encoding (advanced topic).